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The purpose was to test the effectiveness of a $5.00 
incentive on the rate of return of mail questionnaires sent to 353 
schools in the United States. The effects of the geographical region 
and the size of the city in which the sampled schools were located^ 
and the school type (junior high vs. senior high) also were studied. 
Results showed that a $5.00 incentive did not produce actual 
participation from a greater percent of the sample than would be 
obtained if no incentive were offered^ that neither the type of 
school nor size of city caused a differential participation rate, and 
that the percent of schools participating in the evaluation and 
providing incomplete responses varied across geographical regions. 
Greater actual participation once an agreement to participate had 
been obtained was found for schools offered the $5.00 incentive than 
for those offered no incentive^ but the actual gain in data was not 
large. (DT) 
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•xan ri^ECif ':\::\PT) la^ci-sC'vLK data cai;*' tvi-u;" 

Univcr^ifv of Minnesota 
Lc-r[::-C";lc: vr^-ovr^h cv }v^^'jc\\ stiKlics .cf t:Gn reiy on r-^-^Tec^ 

■Jciua ^rc o'tT'm GVzbjcct: r.o crit:^ci^^i;s. As Tarecisi (1950) ':r«;te, ''l-osl 
;\,jLl quusui orJl.:g 60 Zc\: leturn^, m.:' fro.:. :;u;.h a hlpha^ 

S'>-iccl-:ci Mlrttjc:;, thai, the firJ-;r'i*s of such s«.:rvcys pxo nl::4or>u 
invarif.My c-oc-n to rui>i-Lion," because reLui:*! r^t:es arc saldcra abcw 
GO ^c/ci^ni".; Kcrl:I%;:c:;: (iSf.-') r:i-,.'^c«--"-*^s 'chr.t tha rc.^cr.rch^r nvclc urinj; 

cvx^Liorr.oircs. Nevertheless; the mai^lod ql5GGtio:v.^^ire r;::r.:;iriS 
l:hc o::.ly f.:.i:,ible r:-r-a::s of contacf.ing Ir.rrs uunbcrs of per vcns vho 
tivc. scattered ovco: ^ vide geoj^raphic area* ConcequenUlyj inary in- 
ce:ic.>yo recririic»vki.-.c havis L-^c?n tried In ni^tcv.]}i:> tc iruprovc the ruj:):or.be 
rrsues* 

Variables for v:hich published reso-Tch data are available 5.nclede:: 
♦•he ^^.foQtivei:e,i-r^ cf IneJudins a stai^iped return ervelopp (Sccttj 1961. 
GuJJahorn, 1963), cfificial sponsorship (Scott j 1961), lens'::h of quc-c-ti^iv- 
naire (Crr Ke>..;an, 196?-), clacG cf r-ail, i.e., first or third, by uh?ch 
the questicnztaire v:as s.^nt (Cv.ilir/ncrn & Ciillahorn, 3 963), color of que?-- 
tJonr.a^rc nnd ii>rl\i<;ion of novelty iteir.r- wiuh the que?t.ionnnxrs? (Pucei, 
?ieIj;o]i 5. Vheeler. 1970) > dvid perr^onally typed vs. miraccgrc^^phcci cover 
letters (Ar.drcrjron> 19/0, Siipon, 19G7) . 
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Most: research on moneCary incentives has bec^n done for business 
and marketing concerns. Although incentives of lesrS than 25 cents 
appear to be ineffective (Kephart [< Rresslcr, 1958, UatrulM, 1966, 
Eevis, 1948), the 25 ccnn incentive did produce a significantly larger 
response (Kephiirt f< Bressler, 1958) than no incentive. It is inter- 
esting to note, however, in that study the 25 cents v;as not better 
than a follow-up letter. Uatson (1965) in a mail questionnaire study 
for Business VJeek obtained a A8 percent return rat^ vhen lie 
included 25 cents with a short questionnaire, and obtained a ';6 percent 
return rate when he used a follov7-up letter. By combining the tv:o, 
i.e., putting 25 cents in with the questionnaire then sending a follov;- 
up letter tvo days later, he achieved an 84 percent return rate. No 
study was found which used a questionnaire that required a half hour 
or more to complete and none included a large incentive, for example 
$5.00. 

The National Science Foundation evaluation, uithin which this study 
took place, used mailed questionnaires to obtain son-e of its data, and 
incorporated a number of the above techniques in an attempt to obtain 
an adequate response rate. Because mailed instruments were to be sent 
to persons who could gain nothing by their participation, and because 
participants x-zould be required to spend considerable time completing 
several instruments, the use of a monetary incentive appeared particu- 
larly appropriate. 

It was proposed that a monetary incentive might: (1) induce a 
greater number of schools to agree to participate; (2) produce a 
higher response rate (actt^nl part Jcipatlon rate) from those schools 



that agreed to participate; and (3) afCect the quality of returned 
instruments, i»e,, result in higher percentage of returned packets 
with all instruments properly coir.pleted. However, tlie anticipated 
expense of a substantial incentive \:as large and evidence in support 
of such an incentive was meager. Consequently, tlie decision was 
made to tost the effectiveness of a $b incentive before putting the 
technique to extensive use. 

Design and Sairipling 

In addition to the effect of a $5 incentive, the general NSF 
evaluation design provided an opportunity to study the effects of 
three other variables: the geographical region of the United States 
within vjhich sampled schools were located; the size of the city 
within which sampled schools were located; and the type of school 
sampled* These varial>les were combined into a2x5x3x2 fully 
crossed factorial design. 

Tlie n^onetary incentive involved an offer of' $5 each to the 
principal and teacher of a random one-half of the schools (the exper- 
imental group) and offering no incentive to the other half (the 
control group). The experimental group was treated differently from 
the control group in two respects. First, contained in the initial 
letter to schools of the experimental group was the sentence, "We 
are offering $5 to each principal and each teacher v;ho assists in 
this study." Second, two $5 checks, one to the principal and one 
to the teacher, were enclosed in each instrunent packet sent to the 
experimental group schools. 
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The number of levels and the cliaractoristics of the other three 
factors, region, city size, and type of tichool, vjcre dGtormined by 
the needf: of the NSF evaluation. The five regions included: (1) 
Southern California, (2) Southern Indiana and all but the South- 
western part of Michigan, (3) all of Alabama except Birmingham, (4) 
all of North Dakota and ir.ost of Minnesota and Iowa, and (5) about 
one-half of each of the states of Nebraska, Colorado, Utah, Idaho, 
and Montana. (Welch & Gullickson, 1972^ provides a more definitive 
breakdown of the sample.) T;^?o types of schools were involvetl, junioi 
high schools and senior high schools. City size was stratified into 
three categories: (I) cities of population greater than 50,000, 
(2) cities larger than 10^000 but less than 50,000, and (3) cities 
and towns V7ith populations less than 10,000. 

Procedure 

The evaluation concerns vjhich dictated the number of schools selected 
for each factor and each strata are described elsewhere (see Gullickson 
Welch,' 1972). To sample schools, a method called systematic sampling, 
(Cochran, 1953) was used. The technique of systematic sampling, though 
not an entirely random procedure, does produce a sample with the char- 
acteristics of a random sample vjhen the primary sampling units are listed 
in a nonsystematic manner, as was done in this case. Altogether, 353 
schools were sampled. 

A letter requesting the participation of the principal, a teacher, 
and one class of students V7as sent to each school in the sample. The 
letter stipulated that the instruments to be sent would require approx-- 
imately 45 minutes per i)erson to complete, was personally addressed to 
the school's principal, was on NSP stationery, and vjas hand-signed by 

er|c 
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the Director of the NSF Academic Year Study Progranu IVo weeks later, 
a follow-up reminder letter v/ith a second postcard was sent to all 
principals who had not yet responded. 

Instrument packets were then ipailed to all participating schools. 
The mailed instrument packets contained a set of instruinents> specific 
directions for handling the instruments, and a stamped addressed Jiffy 
bag for return of the completed instruments* Each 'teacher and admin- 
istrator was asked co complete a questionnaire, the Science Process 
Inventory (VJelch, 1966) and the Science Attitude Inventory (Moore, 1967). 
The teacher was also asked to administer a set of attitude and achieve- 
ment instruments to a designated class. 

Four weeks after the packets were mailed, a reminder letter was 
sent to all schools v7ho had not as yet returned the completed packets. 
Enclosed with the reminder letter v?as a stamped postcard to be returned 
in the event the principal had not received the instrument packet. Each 
principal that had still not returned the completed instruments tv/o and 
one-half weeks after the reminder letters v/ere sent was then telephoned 
and asked to promptly complete and return the information. 

Results 

Analysis of the data by Ul'IST 570 (Anderson & Frisch, 1971) allowed 
an anova solution of the four indepetident variables. Specifically a 
weighted means (least squares) analysis of city size and an unvjeighted 
means anrilysls of Incentive, Regions, and Schools was carried out. First, 
data acrosiJ all schools sampled was analy?.ed for the tv;o dependent variabler?, 
agreement to participate and actual participation. As can be seen in 
Table 3, cffoctii for Region on the dependent variable. Actual Participation, 
and A x B x G >: D effects for hoth dependent variables were obtained. 



Second, the dependcnl: variables, Actual Participation and Incomplete 
Response, were analysed for the group of schools that agreed l:o participate • 
The results in Table h show: (1) an incentive effect as well as several 
interaction effects for Actual Participation, and (2) a Region effect and 
a Region by city size interaction effect for Incoiriplete Response. Hsu 
& Feldt (1969) and Lunney (1970) tested the applicability of anova 
for dichotomous data, and both indicate the anova is robust, particularly 
where the sample size per treatment level is large (n>50). Both studies 
apply to designs having equal cell sizes and Hsu and Fe?i.dt caution that 
factors in designs having unequal cell sizes would be subject to signifi-- 
cance level concerns caused by possible heterogeneity of variances. 

That caution seens v^ell adv.isecl when interpreting this study *s inter- 
action effects. All significant interactions involved v?idely different 
n*s and because they all included the Region variable, n*s of much less 
than 50 occurred in every such inteiraction. Graphical analyses of the 
interactions suggest that the effects appear to be due to .snicill dJfferences 
across the five regions rather than to trends of any ki^^^d. Mean data is 
provided in Table 5 so that persons desiring may reach tlieir own conclusions 
regarding the two way interactions. 

Insert Tables 1-5 about here 

This study's results do not support the hypothesis that a monetary 
incentive v/ill improve mailed questionnaire response rates. There was no 
indication that a promise of $5 caused persons to agree to participate 
where they otherwise would not have done so (sec Table 3). In fact, obtained 
mean values in Tabic i suggc-sl th-- u».;A>.>Ut- Any beneficial differences 



TABLE 1 

Mean Values for Agreeav^-inl* to Participate and 
Actual Participation for All Schools [Jan^pled 



Agreement to Actual 
Sample Size. Participate Participation 
Source of Variation n X Y 



Region 

Soutliern California 66 ,62 .59 

Indiana-Michic;an 70 .67 . 67 

Alabania 74 .54 .50 

Midwest 74 .77 .73 

Rocky Mountain Area 69 .72 .72 

School 

Junior High 142 .63 .61 

Senior High 211 .69 .66 

City Strata 

1 91 .62 .60 

2 79 .75 .72 

3 183 .66 .63 

Incentive 

Yes 185 .64 .6^ 

No 168 .69 .65 
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TABLE 2 



Mean Values for Actual ParCicipation and Incomplete Ue 


sponsc 


for Sampled 


Schools that Agreed 


to Participate 








Actual 


Incomploto 




Sample Size 


Participation 


Response 


Source of Variation 

_. 


n 


X 


I 


Region 








Southern California 


Al 


.95 


.34 


Indiana-Michigan 


47 


1.00 




Alabama 


40 


.93 


OA 

.20 


Kidv7est 


57 


.95 


. 28 


Rocky Mountain Area 


50 


1.00 


.44 


School 








Junior High 


90 


.97 


.38 


Senior High 


145 


.97 


.31 


City Strata 








1 


56 


.98 


.39 


2 


59 


.97 


.25 


3 


120 


.96 


.35 


Incentive 








Yes 


119 


.99 


.29 


No 


116 


.9/i 


.39 



TABLi; 3 



Analysis of Variance of Rate of A(;rGGment to Participate and Rate 
of Actual Participation for All Schools Sampled 



1 

Source oi 
Vairiation 


df 


Agreement 
To Participate 
MS F 


Actual 
Participation 
MS P 


A (Incentive) 


1 


.322 


1 .49 


. 121 


.55 


B (Region) 


A 


.469 


2.16 


. 600 


2 . 70'' 


C (City size) 


2 


.513 


2.37 


.390 


1 .75 


D (Schools) 


I 


. i-'iS 


.68 


. 187 


.84 


A X B 


4 


.288 


1.33 


.297 


1.34 


A X C 


2 


.019 


.09 


.020 


.09 


A X D 


1 


.223 


1.03 


.399 


1.79 


B X C 


8 


.053 


.24 


.099 


.44 


B X D 


4 


.052 


.24 


.026 


.12 


C X D 


2 


..185 


.85 


.183 


.82 


A X B X C 


8 


.321 


1.48 


.392 


1.76 


A X B X D 


4 


.119 


.55 


.222 


1.00 


A X C X D 


2 


.040 


.18 


.114 


.51 


B X C X D 


8 


.140 


.65 


.099 


.44 


A X B X C X D 


8 


.497 


2.29* 


.496 


2.23' 


Error 


293 






.223 





*p<.05 
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TABLE ^1 



Ana3ysis of Variance of; Rate of Actual rarticxpalion and Rnl:e of 
IncoiTiplGUO Response xor Sanpjed Schools that Agrocd to Parliicipatc 



Actual • Incomplete 

Source of Particrlp jition Response 

Variation df MS P MS y 



A (Incentive) 


1 


.132 




./i28 


2.08 


B (Region) 




.067 


2.26 


.563 


2.74* 


C (City Size) 


2 


.025 


.84 


.145 


.71 


D (Schools) 


1 


.016 


.52 


.033 


.16 


A >: B 


/i 


.088 


2.95" 


.189 


.92 


A >: C 


2 


.081 


2.72 


.071 


.35 


A X D 


1 


.003 


.09 


.213 


1.04 


B X C 


8 


.026 


.86 


.623 


3.03"'" 


B X D 


/i 


.075 




.035 


.17 


C X D 


2 


..008 


.28 


.143 


.70 


A X B X C 


8 


.017 


.57 


.043 


.21 


A X B X D 


/i 


.123 


4.13 


.46) 


2.24 


A X C X D 


2 


.038 


1.27 


.327 


1.59 


B X C X D 


8 


.066 


2.21 


.076 


.37 


A X B X C X D 


8 


.313 


10. s"* 


.387 


1.88 


Error 


175 











'p<.05 
%<.01 
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TAr,Li: 5 

Mean Values of the Significant A k D >: B, and C x B Interactions 
for SaiDplcd Schools that Agreed to Participate 



Actual Participation 



TreDtraent School 



Incentive No Incentive vTr« High Sr, HJgh 

Jlegion n Mean n Mean n Mean n Mean 



Southcni California 


19 


1.00 


22 


.91 


IS 


1. 00 


23 


.9] 


Indiana - Michigan 


21 


1.00 


26 


1.00 


Ik 


1.00 


23 


1.00 


Alabama 


21 


1.00 


19 


.8-4 


13 


.85 


11 


.96 


Midwest: 


29 


.97 


28 


.93 


18 


.9/1 


39 


.95 


UocKy Mountain Area 


29 


1.00 


21 


1.00 


17 


1.00 


33 


1.00 



Incomplete Response 



City Si:^e 

One IVo Three 

Region n Mean n Mean n ' Mean 



Southern California 12 .33 12 .42 17 .29 

Indiana - Michigan 16 .62 11 0.00 20 .45 

Alabama 3 0.00 13 .15 24 .25 

Mf.dwest 9 .11 9 .56 39 .26 

Rocl<v Mountain Area 16 .44 14 .21 20 .60 



due to the incentive must occurred at the tiiT:e or after the partici- 

pants received ihe money (sec Table*: 2 and 6), 

Results as recorded im Trible 3 lead to these conclusions: (1) A $5 
incentive cannot be u>;pccte.d to produce actual participation from a greater 
percent of the sa^r^-ple. than would be obtained if no incentive vjere offered, 
(2) neitlicr the type of school (junior vs. senior hi&b.) nnr the school 
setting (sixe. of the city v7ithxn which a school is located) can be expected 
to caui>e a differential participation rate, and (3) if schools are sampled 
f.rciR the five regions described here, the percent of schools particlpaLin^ 
in an evaluation can be expected to vary across geographical regions. Also 
as can be seen in Table 4 the percent of schools providing incomplete re- 
sponses can be expected to vary across regions. The study provides no 
ansv;crs to vhy the regions differ, but the differences appear great enough 
(Tabic ]) to varrant consideration if sampling is to be carried out across 
those regions of the United States, 

Tables 2 and 4 sur.iinarize participation information of schools that 
agreed to participate. The Incentive Effect for Actual Participation (Tab! 
4) establishes that greater school participation can be expected where $5 
per teacher and principal participant is given, than where no incentive 
has been proffered. Such a conclusion suggests an incentive may better 
insure actual participation once an agreement to participate has been ob- 
tained, Hov7f?.vcr, the actual gain in data v7as not large (see Table 2) and 
suggests that the benefits of increunsed data may not be V70rth the cost. 
Certainly, the KSF evalnatSori project v;ill not incorporate a §5 Incentive 
in its designs for future data gathciring. 
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T\co additioiinl rcr»ultSj avniiablc but not readily apparent in Table 
2, nay have practical implications for iai-ge-scale data gatheiring. A 
concern of tlie evaluation v?as to obtain a large nn:ount of information per 
school. As a result, tlie packets to be sent verc bulky, e>:pensive to 
put together, and expensive to n:ail (total postal charges averaged over 
$2 per packet). Because of the expense of wailing packets, the evaluation 
team first v?rc>te and asked schools for their agreeraent to participate. 
It war; hoped but not kno^^n that uost v/ho responaed affirmatively would 
actually participate v;hen they received the packet. As can be seen from 
Table 2, 97 percent of those agreeing did participate. That figure bodes 
x-jell for others v:ho rre constrained to ask before they niail. 

A second concern evidenced in Table 2 is the high incidence of partial 
response. In nearly a third of the participating schools, eitl-er the prin- 
cipal or the teacher failed to complete all assigned instruments (all stu- 
dent instruments V7ere completed). The o.valuation team resolved the problem 
by returning uncompleted instruments to the responsible persons. The 
importanua of their full participation was explained and they were asked 
to complete the necessary instrument (s) and return iu via the enclosed 
stamped envelope. 

It is not difficult to visualize the seriousness of the problem. If 
the instruments could not (or would not) have been returned for completion, 
t'ne large amount of missing data coupled vith an overall 65 percent return 
rate v;ould have made it impossible to reach more than very tentative con- 
clusions regarding tlie regions sampled. Certainly, tliese results shov: that 
if much data is to be collected from each unit snmpJed, extra precautions 
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must he taken to assure that <l11 instrunicnts \^±11 initially be returned 
properly completed. Failinf, that, resources and schedules nmst be planned 
so that back-up measures can be taken to assure that all participants 
properly conipiete the nscesaary instruments. 

It may be argued that otlier incentives, such as KSF*s active support 
of the evaluation^ masked the positive effect of the monetary incentive. 
That explanation would be move, attractive if the participation rate had 
been quite high, say 85 percent to 90 percent. However, even if it vzere 
true, the argunient provides far better support for incentives othet titan 
money than it does for the monetary incentive. 

It is the opinion of this v^riter, that monetary incentives show little 
promise for being a cost effective method of data collection via mailed 
questionnaires. Certainly the problem of how to obtain a high response 
rate from mailed questionnaires has not been resolved. However, unless 
and until a viable alternative is' obtained, mailed questionnaires V7ill 
remain a necessity for many studies. Therefore, potential solutions must 
be investigated. Possibly an investigation of the effectiveness of 
preliminary contact (s) with subjects coupled with stringent follow-up 
procedures would bo an appropriate direction to turn. 
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